Method for improved detection of nodules in medical images

ABSTRACT

Special displays of medical images are generated that help a radiologist better detect nodules by exploiting the inherent human visualization abilities of detection of symmetry, asymmetry, motion and relative motion. A Region of Interest (ROI) on an x-ray (mammogram, CT-scan, MRI, or other medical image) is selected and the data within the ROI is copied such that a new display now has two or more copies of the information within the ROI. This copy may contain the same relative positions as the original image, or may be a mirror image of the original data. The ROI is moved over the medical image in search of nodules. This movement can be random, under the direct control of the radiologist, systematically moved by a computer algorithm, or some combination of the above.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a continuation of U.S. application Ser. No.14/159,144 entitled METHOD FOR IMPROVED DETECTION OF NODULES IN MEDICALIMAGES, by Tracy J. Stark and Daniel J. Ferlic, which claims the benefitof Provisional Application Ser. No. 61/754,800, filed Jan. 21, 2013,entitled METHOD FOR IMPROVED DETECTION OF NODULES IN MEDICAL IMAGES, byTracy J. Stark and Daniel J. Ferlic. U.S. Provisional Application Ser.No. 61/754,800 and U.S. application Ser. No. 14/159,144 are incorporatedby reference herein.

BACKGROUND

Lung cancer is the second most common cancer among both men and women.With an overall mortality rate of about 90%, it is the leading cause forcancer deaths for both sexes. Survival from lung cancer is directlyrelated to its size when it is detected. The earlier the detection is,the higher the chances of successful treatment are. The cure rate forStage I lung cancers is 70-80%. By the time the nodules grow largeenough for them to interfere with a patient's breathing, then the timefor optimum treatment has passed.

These types of cancers can be detected using medical imaging such asX-rays and CT-scans. They appear as “nodules”, bright circularabnormalities in the X-ray image. Nodules are normally denser than theirsurrounding tissues. They can vary in size, intensity and contrastlevels. On the x-ray they can appear to be located between ribs, on topof ribs, or behind an organ such as the heart.

Most lung nodules do not have any symptoms and are found “accidentally”when a chest x-ray is done for some other reason. When present, thesenodules represent cancer about 40% of the time. This percentage ishigher in those that are at high risk for lung cancer. The larger thenodules are, the easier they are to detect, and the higher theprobability they are due to cancer. The smaller the nodules are, theharder they are to detect. 1 cm is close to the current detection limit.

There are a few diagnostic characteristics that can be used to determineif a nodule is cancerous or not. The larger the nodule, the more likelyit is cancerous. Cancerous nodules on average double in size in about 4months, whereas benign nodules have little or no size change in the sameperiod. Therefore, nodules that grow rapidly are most likely cancerous.The rougher their edges, or the larger they are, the more likely theyare to be cancerous. Smooth, round nodules are more likely to be benign,whereas irregular or “spiculated” nodules are more likely to becancerous. Lung nodules that are calcified are more likely to be benign.Also, lung nodules described as “cavitary,” meaning that the interiorpart of the nodule appears darker on x-rays, are more likely to bebenign.

It is estimated that somewhere between 10 to 20% or more of cancerouschest X-ray nodules are missed on the first reading of the film. Thesmaller the nodule, the higher the percentage of them are missed. When asecond X-ray is taken, and the nodule has grown, then looking back atthe earlier X-rays it is normally obvious that the nodule was present,but was missed by the radiologist.

The early detection and diagnosis of pulmonary nodules in chest X-rayimage are among the most challenging clinical tasks performed byradiologists. They may or may not seek the aid of methods to digitallyenhance the x-rays or automatically detect potential nodules. In testx-rays, current automatic detection methods typically do not identifyall known nodules. If their parameters are set such that they do detectall known nodules, then they also will detect many nodules that arefalse (false positives). How to reduce the number of false positiveswhile maintaining a high true positive detection rate is the mostimportant work in realizing a chest CAD system.

It is apparent that a method to accurately and routinely detect smallerand/or a higher percentage of nodules in chest X-rays, CT-scans, ormammograms has the potential to save lives. We want to detect them whenthey are small since they can grow so quickly, and that chest x-rays arenot taken that often. A lung cancer screening method that can safely andeconomically detect a large number of potentially curable Stage I lungcancers would be an important public health development. The purpose ofthis invention is to provide such a technique.

It has been observed that the human visual system in most individuals isvery good at detecting symmetry and variations from symmetry within ascene. It is also good at detection of motion and relative motion ofobjects within a scene.

If you drop a small object on the floor, you might pick up a similarobject and drop it as well, watching and listening as it falls. Bydropping the known object, we can see (and hear) how it falls, andunderstand what it looks like against the floor background. Having thisreference, and/or the symmetry of two objects on the floor helps inlocating the first object. One of the aspects of this invention is totake advantage of the human visual system to see symmetry, asymmetry,and duplicate objects.

The advertising industry takes advantage of the human visualizationsystem's ability to detect subtle and differential motion. For example,in a TV advertisement they will slowly enlarge and move text that theywant the viewer to pay the most attention to. We can't help but see themovement, and our attention is drawn to the area of the display wherethe movement occurs. Differential movement at the same location in adisplay appears to be more powerful in attracting our attention thanjust simple movement of one object. This is another feature of the humanvisualization system that the current invention can capitalize on.

SUMMARY

This invention attempts to capitalize or exploit our inherent humanvisualization abilities of detection of symmetry, asymmetry, motion andrelative motion by generating special displays that will help aradiologist better detect nodules. This application will focus on theapplication of these display techniques to chest x-rays. However, themethod can also be applied to mammograms, CT-scans, MRI and othermedical imaging modalities.

A Region of Interest (ROI) on an x-ray (mammogram, CT-scan, MRI, etc) isselected and the data within the ROI copied such that a new display nowhas two or more copies of the information within the ROI. This copy maycontain the same relative positions as the original image, or may be amirror image of the original data.

In the current investigations using this method, a rectangular ROI isoptimum, but is not a requirement of the invention. Displaying the copyof the data as a mirror image using a “flat” mirror produces the bestresult. Again the data could just be copied, or a curved mirror could beused, but with our current investigations these will not produce as goodof results. A vertical ROI approximately 1 inch wide is a good defaultthat can be overridden by the radiologist.

The ROI is moved over the x-ray in search of nodules. This movement canbe random, under the direct control of the radiologist, systematicallymoved by a computer algorithm, or some combination of the above. Thesystematic movement by a computer algorithm, which can be overridden bythe radiologist, appears to be the best use of this technique.

Initially moving a thin vertical rectangular ROI (with a single mirrorsurface) systematically from left to right, and then reversing thedirection of motion when the edge of the x-ray is reached appears to bea good default motion. If the mirror is on the right side of the ROI andthe ROI is moving from left to right, the nodules will appear to growfrom the center of the display, under go “cell division”, and then movehorizontally until they leave the outer edges of the ROI. Anatomicalstructures appear to move along non-horizontal paths. When the directionof the ROI movement is reversed, the nodules will appear at each edge ofthe ROI, and move along a horizontal line exhibiting what could becalled “cell collision” or “cell fusion”.

If the mirror surface is on the left side of the ROI, then “cellcollision” will occur when moving from left to right, and cell divisionwill occur when moving from right to left.

The ROI needs to be moved a small amount (relative to both the size ofthe nodule and the width of the ROI), in a short period of time, foroptimum “cell collision” or “cell division” visual effect. Such adisplay method could be considered a movie loop type of animation.

The invention does not require a movie loop type of animation, but someof the best results are obtained when used in this manner. Theradiologist could change the position of the ROI in larger steps and usea larger time delay between frames (displays). This would be analogousto a slide show type of presentation. Each new ROI position could beunder program control, or user control by clicking the mouse forexample.

When changing the position, it is best to have some overlap between thetwo images. With the movie method, the overlap is the entire ROI exceptfor one or two pixels. In a slide show method, the overlap may be forexample ¼, ⅓, or ½ of the width of the ROI in the direction of motion.

Using a slide show type of movement, the symmetry and changes insymmetry are what provide ability to better detect the nodules. Whenusing a movie mode type of movement, both the symmetry of the nodulesand the apparent relative motion of the nodules in relationship to theanatomical features increases the human ability to detect the nodules.

The phrase “anatomical features” refers to such things as blood vessels,bronchial, bones and organs. Circular nodules will have an apparentmotion that is perpendicular to the mirror. Other anatomical featureswill appear to move in a direction that is related to the angle at whichthey intersect the mirror.

Note in the above the use of the verb “appears”. This motion observed isa type of visual or optical illusion due to the movement of the mirrorplane.

The figures will illustrate a ROI as a vertical strip, however thisshould not limit the scope of the invention, but only provide oneexample. The method applies the same for using a horizontal strip, astrip at an arbitrary orientation, a rectangular box that is not thesame size as either dimension of the X-ray or some other arbitraryshape.

The method can also be used on data that has undergone some sort ofmodification, such as digital filtering. For instance, the applicationof a local amplitude normalization, or automatic gain control has beenobserved to improve the ability of CAD systems in detecting nodules.This or some other type of image refinement can be applied prior togenerating the duplicate images. Applying filters that enhance circularfeatures and reduce linear features would also be useful.

The invention is best implemented using a computer program and ahigh-resolution display system. The computer program can be written inany of the currently available languages such as C, C++, C#, or Java. Aplug-in to the ImageJ software (http://rsbweb.nih.gov/ij/) was writtenin Java to provide a prototype of the invention. This prototype was usedto generate the figures used in this patent application.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a state of the art chest x-ray display method.

FIG. 2 illustrates a region of interest denoted on a chest x-ray.

FIG. 3 illustrates a display using one embodiment of the invention.

FIG. 4 illustrates a display using another embodiment of the invention.

FIG. 5 illustrates a few standard user interface elements.

FIG. 6 illustrates the user interface used in a prototype program thatimplements many aspects of this invention.

FIG. 7 illustrates a display of the invention using replication insteadof mirroring the data in the region of interest.

FIG. 8 illustrates using a data refinement method to improve the detectability of potential nodules.

FIG. 9 illustrates changing the padding around the region of interestdisplay.

FIG. 10 illustrates changing the color of the padding locations used inthe display.

FIG. 11 illustrated inverting the color table used to display the x-raydata.

FIG. 12 illustrates using the refine method in conjunction with theinverted color table.

FIG. 13 illustrates changing the size of the region of interest whenusing the refine option.

FIG. 14 illustrates using two mirror surfaces instead of one with refineon.

FIG. 15 illustrates using two replication surfaces with refine on.

FIG. 16 illustrates using two replication surfaces with refine off.

FIG. 17 illustrates using two mirror surfaces with refine off.

FIG. 18 illustrates displaying just the region of interest with refineon and no mirror or replication surfaces.

FIG. 19 illustrates moving the mirror surface across the x-ray to see anodule undergoing “cell division”, without the refine option being on.

FIG. 20 illustrates moving the mirror surface across the x-ray to see anodule undergoing “cell division”, with the refine option being on.

FIG. 21 illustrates how this invention might fit into the workflow ordata stream of a conventional digital radiograph interpretation.

FIG. 22 illustrates the various computer components that this inventionmight utilize.

DETAILED DESCRIPTION

The following detailed descriptions can be read in connection with theaccompanying drawings in which like numerals designate like elements andin which:

FIG. 1 illustrates a state of the art chest x-ray display method. Thechest x-ray 110 is displayed inside of a window frame 100. In thisexample the public domain program ImageJ (http://rsbweb.nih.gov/ij/) wasutilized. The program has a user interface system that is not shown.Radiologists will look at digital chest x-rays 110 using a program suchas this. Film x-rays are displayed using a light box. This inventionconcerns a method of display for digital x-rays. Film x-rays can bedigitized to utilize this invention.

FIG. 2 illustrates a Region Of Interest (ROI) 220 denoted on a chestx-ray 110. In this display the ROI is marked with a box 210. The linesize, line style, opacity, color and other graphic attributes of thisROI box 210 is something that could be changed by the user. Marking theROI box 210 on the x-ray 110 is not a requirement of the invention, butdoing so is convenient and helps the radiologists orient themselves. TheROI 220 shape can either use a program default, or be defined by theradiologist. In this case, the ROI 220 is a vertical rectangle of widthW and height H. The optimum shape of the ROI 220 is either a verticalrectangle as displayed here, or a horizontal rectangle in which H wouldbe about the same value as the current W, and W would be such that itextends the width of the x-ray. A symmetry or replication axis 230 isassociated with the ROI 220. In the case of a mirror type display, 230will represent a symmetry axis. If the data are not mirrored, then 230will be considered a replication axis. Normally 230 will be a straightline and be along either of the longest sides of the rectangular ROI220. If the ROI 220 is some arbitrary shape then the replication axiswill be a portion of the irregular shape boundary. It is anticipatedthat using a non-rectangular (or square) shape for the ROI 220 willproduce sub-optimal results. The default value for W could be set to thenumber of pixels closest to 1 inch of x-ray. The default value for Hcould be set to the height of the x-ray.

FIG. 3 illustrates a display using one embodiment of the invention. Thisdisplay is contained in the window 300. This is a separate window fromwindow 100 that contains the x-ray 110 and ROI outline 220 shown in FIG.2. The data 310 of the display 300 contains the data from the ROI 220and a copy of the ROI data 320, along with left 330 and right 340padding.

In FIG. 3 the left 330 and right 340 padding are of equal width and areequal to twice the width W (seen in FIG. 2) of the ROI. These are gooddefault values. For the sake of symmetry, it is nice to have the leftpad width 330 and right pad width 340 to be the same, but the inventionwill still function if they are of different values. Also the inventionwill function if their widths are made either larger or smaller thantwice the width W (seen in FIG. 2).

The dashed line 350 in this figure marks the axis of symmetry. It isshown here as an example, but normally would not be included in adisplay used by a radiologist to look for nodules. The user interfacecould contain a toggle to turn the line on and off. The default settingwould be to have it turned off. Line 350 is an axis of symmetry becausethe data of 320 is a mirror image of that contained in 220. If the datain 320 were the same orientation as that of 220, dashed line 350 woulddenote a line of replication.

In this embodiment of the invention, the position of the axis ofsymmetry 350 will not change inside of the window 300 as the ROI 220 ismoved about the x-ray data 110 shown in FIG. 2. The data of the x-raywill appear to move relative to the position of the dashed line 350.

The dash oval 360 is used to denote a potential nodule. The radiologist,using a mouse or other user interface implementation, could draw such anoval to record the position of the nodule. If the display were on aniPad, (or other touch screen device) the radiologist could just put hisor her finger on the nodule to record its location.

This window 300 would need some sort of user interface. Such a userinterface could be similar to 420 shown in FIG. 4. It could be attachedto the bottom, side(s), or both, of window 300, or placed in its ownseparate window. Alternatively its functionality could be placed in pulldown menus of the main program. Those versed in the state of the art ofcomputer programming readily know how to create a user interface, so itis not included here. An example of a user interface that might be usedby this invention will be provided in subsequent figures. Such a userinterface is intended for an example only and should not be construed aslimiting the scope of the invention.

FIG. 4 illustrates a display using another embodiment of the invention.The display for this embodiment is in its own window 400 just as theprevious embodiment. However, in this embodiment the ROI and itsreplication(s) are displayed relative to the original position of theROI on the chest x-ray. The data 410 in this display contains padding onthe left 430 and padding on the right 440. The default size of the leftpadding 430 is the same as the padding 330 around the ROI, while thedefault size of the right padding 440 is the sum of the padding 340, theROI 220 and the mirrored (or replicated) data 320.

In this embodiment, the axis of symmetry 350 will traverse across thewindow 400 to mark the relative location of the mirror surface orreplication surface as it is moved. As mentioned for the previousfigure, the graphic to represent the mirror surface 350 can be underuser control. However, normally a line showing the mirror surface wouldnot be displayed. The location is implied by the display itself.

This window 400 has a user interface 420 associated with it. In thisexample the user interface 420 is placed below the data 410. The widgetswithin the user interface 420 are provided as examples of some of thevariables that could be under user control. Not all of the variablesshown need to be under user control, nor are all possible variables thatcould be within the user interface contained within this example of auser interface. The user interface could be in a separate window, placedin pull down menus, exposed by mouse or key click, or hooked to physicaldials and sliders. The example provided here is just an example andshould not be considered to limit the invention.

FIG. 5 illustrates a few standard user interface elements that can beused by this invention. This figure does not illustrate all of the userinterface elements that might be used, but only those that are used inthe following figures produced from the invention prototype. This figureshould not be used to limit the scope of the user interface items thatmight be associated with the invention. Those skilled in the art ofcomputer programming and using current computer user interfaces willknow how to utilize such user interface widgets.

Item 510 is a push button. Clicking on a push button is normally used tocauses the program to perform some predefined action. The push buttonoften has a word associated with it to provide a hint as to what actionwill be performed. In this case “DONE” might signal the program toterminate.

Item 520 refers to a slider bar. Slider bars are used to adjust aparticular value. 524 indicates the title of the slider bar, whichprovides the user a hint as to what the slider bar controls. 528indicates the current numerical value of the slider bar. In someimplementations of a slider bar the user can type a new number in thisfield to change the value of the slider bar parameter. 522 alsoindicates the current value of the slider bar parameter, and can also beused to change the current value. 522 provides a relative value from thestart to the end of the control range. The control range is defined bythe program and may or may not be apparent from the visualimplementation of the slider bar. In this example of a slider bar therange is not apparent. The user can use a mouse or other pointing deviceto change the location of 522, which will then update the value 528.Clicking on the left arrow 526 will decrement the slider value 528 andcause 522 to move to the left by some program-defined increment.Normally the increment is set to one, but this can be any arbitrarynumber. In some cases another user interface element will be provided tochange the increment of a slider bar. Clicking on the right arrow 527will increment the slider value 528 and cause 522 to move to the rightby some program-defined increment. Normally the right and leftincrements are the same.

Items 530 and 540 are check boxes. They normally are used to change thestate of a variable. They contain a check box and a description field.The check box indicates either an “off” status as in 530 or an “on”status as in 540. In 530 the description field contains “Refine” while540 contains “Mirror”. Therefore, in this example the data that aredisplayed are not in the “Refine” state, but are in the “Mirror” state.

FIG. 6 illustrates the user interface 420 used in a prototype programthat implements many aspects of this invention. This user interface iscomposed of three types of widgets: slider bars, buttons and checkboxes. These have been described in general in the discussion of FIG. 5.Here we will discuss what each of these elements control for thisparticular implementation of the invention. This discussion should notbe viewed as limiting the scope of the invention, but providing someexamples of how it might be used and exploited.

In FIG. 4, and in several subsequent figures, this user interface isshown in conjunction with display 400. It could also be used with theembodiment shown in FIG. 3 as well. It could be attached to the displaywindow, or be in its own display window. The layout of the widgets couldbe changed, more added, and some removed to better fit the needs of theRadiologist.

Slider bar 610 controls the “Primary Mirror Location”. The “PrimaryMirror Location” is represented with a dashed line 350 in FIG. 4. Amouse or other pointing device can be used change the location of themirror via this slider bar. The program could also be modified such thatthe movement of the primary mirror location 350 could follow the mouseor other pointing device that is moved about the display 400 of FIG. 4.In this implementation the number associated with the location sliderbar is in number of pixels. It could also be provided in millimeters orinches. For this particular x-ray there are 5 pixels per mm.

Push button 615 is used to signal the program that the radiologist isdone with evaluating this particular x-ray.

Slider bar 620 controls the width (W in FIG. 2) of the ROI. This allowsthe radiologist to see more or less of the x-ray in the mirrored orreplicated displays. For example, it will control the width of the datacontained in 220, 320, 920, 930, 940, and 942 as found in the otherfigures. In this implementation the number associated with the widthslider bar is in number of pixels. It could also be provided inmillimeters or inches. For this particular x-ray there are 5 pixels permm.

Slider bar 625 controls the amount of padding (330 and 340) present oneither side of the ROI and its copies. In this example the amount ofright and left padding is the same. The padding is used to visuallyseparate the mirrored or replicated display from the rest of the x-rayso that the eyes will focus just on the current ROI and its copies. Theamount of padding can be reduced to a small number (even zero) to bettersee the ROI within the context of the x-ray. In this implementation thenumber associated with the pad slider bar is in number of pixels. Itcould also be provided in millimeters or inches. For this particularx-ray there are 5 pixels per mm.

Slider bar 630 is used to change the increment of the primary mirrorlocation used by slider bar 610. This affects how many pixels theprimary mirror location will change each time either the right or leftarrows (i.e., see 526 and 527 of FIG. 5) of slider 610 is clicked. Italso sets the increment used when the mirror location is being changedautomatically when the “Animate” checkbox 675 is on.

Slider bar 635 contains the number of mirror surfaces (symmetrylines/axes) present in the display. The number of replication surfaces(lines/axes) is the same as the number of mirror surfaces (symmetrylines/axes). Whether or not the data are mirrored or replication isdetermined by the “Mirror” checkbox 690. If the number of mirrors is setto 0, then just the ROI 220 is display surrounded by the padding widthdetermined by slider bar 625.

Slider bar 640 contains a time delay between display refreshes. Thelarger the number, the more time the program takes before updating thedisplay. This is only applicable when the “Animate” checkbox 675 is on.On some computers, if the wait value is too small, the display willappear to “flash” because a new image is being displayed before thecomputer was finished calculating the previous image. On other computersthe display many change too quickly and need to be slowed down to graspthe changes.

Push button 645 “Invert” is used to invert the current color lookuptable. FIGS. 11, 12, and 13 provide examples of inverted lookup tables.

Push button 650 “Flip” is used to flip the color of the padding. In thisimplementation the normal padding is black. Pushing this button oncewill change it from black to white, or white to black depending upon itscurrent value. Such a change in state could have been accomplished usinga check box instead of a push button.

Push button 655 “Reverse” is used to change the direction in which theprimary mirror location is moving when the “Animate” check box 675 ison. For example, if it is moving from right to left, pushing this buttonwill reverse the direction to left to right. This button might be usedto better understand whether or not an anomaly observed on the x-rayshould be classified as a nodule.

Push button 660 “1/4” is used to change the value of the incrementslider bar 630 to one quarter of the current value of the width sliderbar 620. This has application when using the right and left arrows ofslider bar 610 to move the mirror (or replicated) display in a slideshow type of manner. In this case, each time the mirror is moved, therewill be a 75% overlap with the previous display.

Push button 665 “1/2” is used to change the value of the incrementslider bar 630 to one half of the current value of the width slider bar620. This has application when using the right and left arrows of sliderbar 610 to move the mirror (or replicated) display in a slide show typeof manner. In this case each time the mirror is moved, there will be a50% overlap with the previous display.

Push button 670 “1/1” is used to change the value of the incrementslider bar 630 to be equal to the current value of the width slider bar620. This has application when using the right and left arrows of sliderbar 610 to move the mirror (or replicated) display in a slide show typeof manner. In this case each time the mirror is move, there will be nooverlap with the previous display.

Check box 675 “Animate” is used to turn the animation of the primarymirror location on and off. The animation starts up in the direction itwas going when it was previously turned off. It will move the mirrorlocation 350 by an amount equal to the current value of the incrementslider bar 630. Its action when it reaches one end of the slider bar isdetermined by the state of the “Swing” check box 680. The “wait value”set with slider 640 determines a relative length of time the programpauses before showing the data from the new mirror location.

Check box 680 “Swing” determines what action is taken when the primarymirror location is incremented past either the right or left end of theslider 610 limits. If the “swing” check box 680 is on, then the motionof the animation will change direction and the slider value will headback in the opposite direction. If the “Swing” check box 680 is off,then when the slider moves past one end of the slider bar, it will jumpto the opposite end of the slider bar.

Check box 685 “Refine” is used to provide some type of data refinementto the displayed x-ray in the ROI and its copies. In this example a dataamplitude normalization algorithm is used. The algorithm is a type ofAutomatic Gain Control (AGC) that might be used for seismic data. TheRefine button could be used for any number of different data refinementalgorithms. This is used just as an example of how modifying the x-rayvalues can improve the detect ability of nodules, particularly whenusing the mirrored or replicated display of this invention.

Check box 690 “Mirror” controls how the data are display across the axisof symmetry 350. If the check box 690 is on, the data are mirroredacross the axis of symmetry 350. If it is off, the data are copied. FIG.7 provides an example display of when check box 690 is off.

FIG. 7 illustrates a display of the invention using replication insteadof mirroring of the data in the region of interest. FIG. 4 and FIG. 7are a comparison of using the symmetry axis 350 to either mirror orreplicate the data in the ROI 220. Note that in FIG. 4 there is a smoothdata transition across 350 going from the data in 220 to that in 320. InFIG. 7 there is a jump in the data continuity across 350 when going fromthe data in 220 to that in 920. Turning the mirroring on and off changesthe separation of the nodule at the top of the ROI. The nodule is easierto detect in FIG. 4, than in FIG.7 due to the better symmetry of thedisplayed data.

FIG. 8 illustrates using a data refinement method to improve thedetectability of potential nodules. In FIG. 8 the “Refine” check box 685has been turned on. This converts the FIG. 4 data 220 and 320 to that of930 and 940 shown in FIG. 8. With this refinement method, the nodulesare much easier to detect since they stand out more from the localbackground values than they do in FIG. 4. This is particularly true whenthe mirror plane 350 is animated in a movie loop fashion.

FIG. 9 illustrates changing the padding around the region of interestdisplay. To generate FIG. 9, the “Pad” slider bar 625 is used to reducethe padding from 250 (in FIG. 8) to 12 (in FIG. 9). All other displayparameters are the same between these two figures. Reducing the paddingallows the radiologist to view the ROI (930 and 940) in the context ofthe entire x-ray. Note that 930 is in the proper position, but 940(mirror image of 930) overlays a portion of the original chest x-ray. Ifthe radiologist wanted to, he/she could have reduced the padding to 0.

FIG. 10 illustrates changing the color of the padding locations used inthe display. FIG. 10 should be compared to FIG. 9. The “flip” button 650was pushed to flip the color of the padding from black (in FIG. 9) towhite (in FIG. 10). In this implementation, the flip button changed thecolor of the padding found in 430, 330, 340 and 440. The purpose ofperforming such a change is mainly aesthetics and radiologist preferenceon whether black, white or some other color helps to distinguish orisolate the ROI 220 and 320 from the rest of the x-ray. A program toimplement the invention does not need to have the ability to change thecolor of the padding, but having such an option should improve theusability of the invention.

FIG. 11 illustrates inverting the color table used to display the x-raydata. FIG. 11 should be compared to FIG. 10. The “Invert” button 645 waspushed to invert the grey scale of the x-ray within the image 410. Notethat in this implementation, inverting the color scale does not changethe padding color. It is still white. In this inverted color scale thenodules will be darker than their surroundings instead of lighter as inthe previous figures. Which color look-up table to use is a matter ofthe radiologist's personal preference. A program to implement theinvention does not need to have the ability to change the x-ray colorscale, but having such an option should improve the usability of theinvention.

FIG. 12 illustrates using the refine method in conjunction with theinverted color table. FIG. 12 should be compared with FIG. 11. Thedifference between these two figures is that the “Refine” option 685 hasbeen turned on in FIG. 12 and not in FIG. 11. In this figure the noduleswill be darker than their surroundings. The nodule towards the top ofthe ROI (as denoted with 360 in FIG. 3 and FIG. 4) is easier to see inFIG. 12 than in FIG. 11.

FIG. 13 illustrates changing the size of the region of interest whenusing the refine option. In FIG. 13, the width of the ROI 930 and itsmirror image 940 has been changed from 125 in FIG. 12 to 250 in FIG. 13.With this increased ROI size the nodule is a little more apparent. Thegeometric patterns of the ribs and bronchioles are very different fromthat of the single nodule.

FIG. 14 illustrates using two mirror surfaces with refine on. FIG. 14 isa change relative to FIG. 9, in which the mirror slider bar 635 is usedto increase the number of mirrors from 1 to 2. This adds the displayregion 942 to the overall Region of Interest ROI display. A new mirrorsurface (otherwise called an axis of symmetry) 352 is inherent in thedisplay. The dashed line 352 shows the location of this new mirrorsurface, but in normal use of the invention the dashed line would not bedisplayed. An arbitrary number of mirrors can be utilized in theinvention. Using an odd number of mirrors generally works better thanusing an even number of mirrors. With an odd number of mirrors, ifnodules are present, they will always appear in pairs.

FIG. 15 illustrates using two replication surfaces with refine on. FIG.15 is a change relative to FIG. 14 where the mirror check box 690 hasbeen turned off. This causes the region marked 940 in FIG. 14 to become941 in FIG. 15. Note that in FIGS. 15 930, 941, and 942 look the same,except they are displaced relative to each other. The x-ray data acrossthe replication axes 350 and 352 is no longer smooth, but showsdiscontinuities. Some radiologist may find the movement of thereplication mirror plane across the x-ray to provide a better means ofidentifying nodules, but it is expected that most radiologist willprefer using the mirror planes, with refine on.

FIG. 16 illustrates using two replication surfaces with refine off. FIG.16 is a change relative to FIG. 15, where the refine checkbox 685 isturned off. Most will find the nodule is a little harder to see in FIG.16 than it is in FIG. 15. This figure further illustrates the importanceof using data refinement methods, in conjunction with the invention, tobetter identify potential cancerous nodules.

FIG. 17 illustrates using two mirror surfaces with refine off. FIG. 17is a change relative to FIG. 16, where the mirror checkbox 690 is turnedback on. This has the affect of reversing the direction in which thedata are displayed in 920 of FIG. 16 to produce 320 in FIG. 17. The datatransitions smoothly across the lines of symmetry 350 and 352 in FIG.17, instead of having discontinuities across these lines in FIG. 16.Most will find the nodule easier to see in FIG. 17 than FIG. 16.

FIG. 18 illustrates displaying just the region of interest with refineon and no mirror or replication surfaces. FIG. 18 is a change relativeto FIG. 9, where the number of mirrors slider bar 635 is changed from 1in 0. In FIG. 18, just the ROI 930 is displayed using the datarefinement method. This feature of removing the mirrored or replicateddata is useful if the radiologist wants to see just the ROI 930 in itscontext of the chest x-ray 410. This can allow the radiologist to betterinsure he/she does not identify a false positive nodule. In this figurethe padding width 625 is small enough to clearly identify the ROI 930without covering much of the background x-ray.

FIG. 19 illustrates moving the mirror surface across the x-ray withoutthe refine option being on. FIG. 19 contains a sequence of five screencaptures of the invention embodiment shown previously in FIG. 3. InFIGS. 19, 700, 710, 720, 730, and 740 represent the display when the ROI(220 of FIG. 2) is moved in steps across the x-ray 110 of FIG. 2. Aparticular area of this display is noted with 750 to observe whathappens when the mirror surface encounters a nodule.

In viewing these images sequentially from left to right, the nodulelooks like it is undergoing “cell division”. If viewed sequentially fromright to left it would look like “cell collision”.

In 700 the axis of symmetry (350 of FIG. 3) is just on top of the leftedge of the nodule. In 710, the axis is about in the middle of thenodule, and it appears as if there is one entire nodule in the display,whereas what we are really seeing is only the left half of the noduleand its mirror image. In 720 the mirror surface has moved close to theright edge of the nodule. This image has the appearance of a cell justabout to complete cell division. In 730 the mirror surface is just tothe right of the nodule, and in 750 it is a little ways past the rightside of the nodule.

When shown in a movie loop, with a much finer movement of 350, it willlook like the nodule grows from the center of the display to a singlenodule, this nodule will then split into two (like a cell dividing), andthen the two nodules will separate from each other and move further andfurther apart along a horizontal line until they leave the display. Onlyfeatures in the x-ray that are circular will exhibit this type ofapparent “cell division” motion. Non-nodule feature will have a verticalcomponent to their apparent motion, so they will appear to either falldown from, or climb up from the mirror surface. When the ROI moves fromright to left, the sequence of images will be reversed. In this case thetwo nodules will come together along a horizontal line, collapse intoone, and then disappear. This type of motion might be called “cellcollision” or “cell fusion”.

FIG. 20 illustrates moving the mirror surface across the x-ray with therefine option being on. This is a similar set of screen captures asshown in FIG. 19, with the exception that the data displayed have hadthe refine method applied to them before being displayed. Thisparticular refinement method improves the detectability of the nodulecontained in the oval marked as 850.

FIG. 20 contains a sequence of five screen captures of the inventionembodiment shown previously in FIG. 3. In FIGS. 20, 800, 810, 820, 830,and 840 represent the display when the ROI (220 of FIG. 2) is moved insteps across the x-ray 110 of FIG. 2. A particular area of this displayis noted with 850 to observe what happens when the mirror surfaceencounters a nodule.

In viewing these images sequentially from left to right, the nodulelooks like it is undergoing “cell division”. If viewed sequentially fromright to left it would look like “cell collision”.

In 800 the axis of symmetry (350 of FIG. 3) is just on top of the leftedge of the nodule. In 810, the axis is about in the middle of thenodule, and it appears as if there is one entire nodule in the display,whereas what we are really seeing is only the left half of the noduleand its mirror image. In 820 the mirror surface has moved close to theright edge of the nodule. This image has the appearance of a cell justabout to complete cell division. In 830 the mirror surface is just tothe right of the nodule, and in 850 it is a little ways past the rightside of the nodule.

When shown in a movie loop, with a much finer movement of 350, it willlook like the nodule grows from the center of the display to a singlenodule, this nodule will then split into two (like a cell dividing), andthen the two nodules will separate from each other and move further andfurther apart along a horizontal line until they leave the display. Onlyfeatures in the x-ray that are circular will exhibit this type ofapparent “cell division” motion. Non-nodule feature will have a verticalcomponent to their apparent motion, so they will appear to either falldown from, or climb up from the mirror surface. When the ROI moves fromright to left, the sequence of images will be reversed. In this case thetwo nodules will come together along a horizontal line, collapse intoone, and then disappear. This type of motion might be called “cellcollision” or “cell fusion”.

FIG. 21 illustrates the main components of a system, 2100, that might beused to implement this invention. In this figure the solid lines witharrows (for example 2101) indicate the primary flow of information ordata. The dashed lines with arrows (for example 2102) indicate thepotential flow of information or data.

The center of FIG. 21, 2150, is the processing of the data as per thisinvention. This box will be explained in more detail with FIG. 22.

There are three inputs into 2150. Item 2110 represents the priormetadata associated with the digital radiographic data that comes from2111. In some systems these two pieces of information may come from thesame source, or they could come from different sources. The priormetadata consists of such things as the patient's name, informationconcerning how the radiographic data were collected, the format the datais store in, prior interpretations or readings of the data, and othertypes of metadata that are currently, or might be in the future,associated with the digital radiographic data or the patient. Item 2111represents the actual digital radiographic data that will be used by theinvention. This data might be in a DICOM, RAW, or some other data formatthat is, or later becomes, standard practice for digital radiographicdata storage. This information may come directly from the x-ray machine,an external storage device such as a CDROM or thumbdrive, via theinternet, an intranet, or other types of digital data storage devices.This information may be stored in PACS (picture archiving andcommunication system), RIS (radiology information), CVIS (cardiovascularimaging systems), or other such systems that are, or may in the future,be used to store and organize radiographic data and its associatedmetadata. Also feeding into 2150 are user commands, 2112, that provideinstructions on how the invention is to be utilized. The methods inwhich these data are fed into 2150 will be discussed more in FIG. 22.

There are three possible outputs from 2150. These are updated metadataassociated with the digital radiographic data (2113). For the most partthis will be the location of nodules identified by this invention, theradiologist's comments concerning the nodules or other aspects of thex-ray, along with other information such as the time, date, location anddoctor's name who performed the diagnosis. Display settings used in theinvention to obtain the results might also be stored. The radiologistmight also decide to keep a copy of any modified radiographic data, suchas that produced by the refine method (e.g. 685 of FIG. 8) that isapplied to all or a portion of the digital radiograph. The radiologistmight also want to generate and save movie loops that illustrate aparticular feature. These might be stored as radiographic data (2114) ormetadata (2113).

The pertinent parts of the radiologist's findings needs to betransmitted to the requesting doctor and/or to the patient directly.This is done in item 2115. Normally these findings would be placed inthe metadata repository, such as a PACS system, and the doctor wouldaccess such a system to obtain the radiologist's findings. However,there may be times in which pertinent findings need to be transmitted tothe doctor and/or patient without delay. Therefore the process shouldhave an avenue of doing this. It could be performed via an emailmessage, text message, or voice message to the appropriate recipient.

FIG. 22 provides an example of a processing system (2150) used toimplement this invention. In this figure the solid black lines, such as2206, indicate how the devices could be connected. These connections maybe a physical wire, or part of the same physical device, being on thesame circuit board, or chip. They could also occur wirelessly. Each ofthe individual items in this figure could be combined with other itemsin the same physical container, or they may be separate. Not all of theitems have to be present. Other items could be connected that are now,or might be in the future, used with a computer processing device.

Such a system (2150) would have means of retrieving input data (2201)from some form of storage device. This might be a physical device suchas a disk drive hooked directly to the computer processing device(2205), an external storage media and media reader such as a CDROM andCDROM reader, or some storage accessed via the internet, local areanetwork, Bluetooth, etc. This device is where the input data (2110 and2111 of FIG. 21) most likely would come from. The user commands, 2112FIG. 21, might also come from such devices, but not normally.

The system also needs some form of output storage device, 2202. Thiscould be the same or similar device as those used for input (2201).

The system will also need some form of display device (2203). This couldbe any type of display device, some example of which are: computermonitor, a video projection system, a tablet, or touch screen device.Hard copy devices such as printers, plotters, and cameras could also beused to display the results of this invention. Any device currentlyused, or that is used in the future, to view the results of a computercould be used as the display device, 2203. There might be more than onedevice. The computer might display the same information on each of themultiple devices, or different information on different devices.

The user will need to interact with the computer through some form ofuser interaction device, 2204. This is predominantly where the usercommands, 2112 of FIG. 21, will be obtained. A keyboard and mouse arecommon types of interaction devices. However a touch sensitive screenand finger, some form of motion tracking, eye tracking or voicerecognition device might also be used. Any currently used device, ordevices used in the future, to provide input to a computer type ofdevice (2205) could be used as the user interaction devices (2204).There may be one or multiple such devices.

The computer processing device, 2205, is at the center of this system.This might be a standard computer, a graphics computer, GPU, some typeof computing tablet, an iPad, a parallel computer, or some form of cloudcomputing. The computer processing device would need to contain memory,some form of operating system, a means of communicating with the otherdevices, and to run or execute the data processing and display softwarewritten for this invention (2208). It would also benefit greatly fromrunning conventional data processing and display software (2207).Devices needed for 2205 are well known by the industry.

The system 2150 could be made of a different physical device for each ofthe elements of FIG. 22, devices that have several of the devices shownin FIG. 22 combined into one device, or something that contains all ofthe devices shown in FIG. 22 in one physical device. For instance, alaptop computer could be used to house all of the elements of FIG. 22into one physical device.

Item 2208 refers to the software written to implement this invention.Using the description provided above, an individual skilled in computerprogramming should be able to easily implement this invention usingcomputer languages such a C, C++, Java, and other know computerlanguages, and those that might be available in the future.

Item 2207 refers to software currently utilized by radiologist to viewand interpret radiographic data. The “iNtution” software from TeraReconis one such software package that might be used by a radiologist tostudy chest x-rays to identify lung nodules. Another package might beOsiriX, an open-source PACS workstation DICOM viewer(www.osirix-viewer.com).

The radiologist will use the User interaction devices (2204) to instructthe program (2208) which options to implement. As the radiologist scansa radiograph with this invention, the radiologist will identify nodulesbased on the radiologist's knowledge of the human anatomy and the typeof radiograph the radiologist is viewing. The invention will help makesuch nodules easier to identify. Once a nodule, or potential nodule isidentified, the radiologist will use the User interaction device, (2204)to instruct the program (2208) to record the location of the nodule(along with any comments the radiologist might want to make concerningthe nodule). Such records can then be stored to the Output data storagemedium (2202). The radiologist will then continue to scan the radiographuntil the radiologist is no longer able to identify additional nodules.In performing such viewing, the radiologist might want to also view ormanipulate the radiograph using conventional data software (2207). Orthe invention might be licensed to makers of conventional dataprocessing software (2207) such that the invention software (2208)becomes a part of the conventional software package (2207).

One useful form of modifying the input data to improve the ability todetect nodules is to apply a gain function that has the form of equation1 below. The variables A, B, and C can be set to produce a variety ofamplitude changes, such as those that are known in the industry, forexample a conventional (AGC). It can also produce new gain effects, suchas the Refine Gain, and Remap Gain methods described below. These newgain methods can help enhance the anomalous x-ray values that may beindicative of nodules. This type of modification changes the values ofthat data that are displayed through the display color lookup table,therefore a test needs to be made that the output value calculate doesnot exceed either the minimum or maximum allowed value. If it does thenit should be set to the appropriate minimum or maximum value it exceeds.

output=(input−A)*B+C   (1)

where:

-   -   output=changed value of an input pixel in the modified image,    -   input=original value of the corresponding input pixel in the        input image,    -   A=a user defined constant, possibly in combination with a value        derived from the pixel values in a window around the input        point,    -   B=a user defined constant possibly in combination with a value        derived from the pixel values in a region around the input        point,    -   C=a user defined constant possibly in combination with a value        derived from the pixel values in a region around the input        point,

No Gain Application:

Equation 1 can be used to leave the input data unchanged by using:

-   -   A=0,    -   B=1, and    -   C=0.

Conventional AGC:

To reproduce a conventional AGC routine,

-   -   A=either zero, the mean or the median of the input image,    -   B=scalar/(local mean), and    -   C=A,    -   where the scalar could be the mean or median value of the        amplitude range used for the display's color lookup table, and        the local mean is calculated from a region around the input data        point.        There are several ways to calculate the local mean.    -   1) calculate either the mean or median of all the pixels located        in a box that is centered on the input point. The size of the        box can be proportional to the width of the strip that is being        mirrored.    -   2) similar as 1) above, but use the same local mean value for        all pixels on the same row when using a vertical strip, or the        same column when using a horizontal strip.

Refine Gain Method

This method in current studies provides better visual discrimination ofnodules than the conventional AGC gain method. For this gain method,

-   -   A=local mean,    -   B=constant gain value (for example 2.5),    -   C=constant gain value (for example 0.9) multiplied by regional        mean (e.g. entire image)

Remap Gain Method

-   -   A=Lmin,    -   B=(Gmax−Gmin)/(Lmax−Lmin)    -   C=Gmin

Where:

-   Lmin=local minimum value around input pixel-   Lmax=local maximum value around input pixel-   Gmin=global minimum value (from entire or large portion of image)-   Gmax=global maximum value (from entire or large portion of image)

In implementing this gain function, the region around the input pixelcould have an area equal to the square of the width of the smallestdimension of the region of interest.

While the invention has been described with reference to an exemplaryembodiment(s), it will be understood by those skilled in the art thatvarious changes may be made and equivalents may be substituted forelements thereof without departing from the scope of the invention. Inaddition, many modifications may be made to adapt a particular situationor material to the teachings of the invention without departing from theessential scope thereof. Therefore, it is intended that the inventionnot be limited to the particular embodiment(s) disclosed, but that theinvention will include all embodiments falling within the scope of theappended claims.

1. A medical imaging system, comprising: a computer system configuredto: (a) receive image data representing a medical image; (b) receive aselection of a region of interest in the medical image; (c) copy theimage data contained within boundaries of the region of interest; (d)output, for display, an output image containing a first copy of theimage data contained within the boundaries of the region of interest anda second copy of the image data contained within the boundaries of theregion of interest so that the first copy of the image data and thesecond copy of the image data are simultaneously displayed in the outputimage and separated by either an axis of symmetry or an axis ofreplication; (e) move the boundaries of the region of interest withrespect to the medical image and repeat steps (c)-(d) so that as theoutput image changes with each movement of the boundaries of the regionof interest, a position of a nodule appearing in the displayed firstcopy of the image data changes in a synchronized manner with respect toa position of the same nodule simultaneously appearing in the displayedsecond copy of the image data; and a display device configured toreceive the output image and display the first copy of the medical imagedata simultaneously with the second copy of the medical image data andseparated by either the axis of symmetry or the axis of replication. 2.The medical imaging system of claim 1, wherein the first copy of theimage data includes original image data contained within the boundariesof the region of interest; and wherein the second copy of the image dataincludes a mirror image based on the image data contained within theboundaries of the region of interest.
 3. The medical imaging system ofclaim 1, wherein the first copy of the image data includes originalimage data contained within the boundaries of the region of interest;and wherein the second copy of the image data includes a replicatedimage based on the image data contained within the boundaries of theregion of interest.
 4. The medical imaging system of claim 1, whereinthe computer system is configured to move the region of interest aseries of times; and wherein the display device is configured to displaythe output image as an animated display based on the series.
 5. Thesystem of claim 1, wherein the computer system is further configured todigitally process the image data with algorithms that enhance thedetectability of nodules.
 6. A medical imaging system, comprising: acomputer system configured to: a) receive image data representing amedical image; b) receive a selection of a region of interest in themedical image; c) copy the image data contained within boundaries of theregion of interest; d) modify the copy of the image data from within theboundaries of the region of interest; e) output, for display, an outputimage containing a first copy of the modified image data from within theboundaries of the region of interest and a second copy of the modifiedimage data from within the boundaries of the region of interest so thatthe first copy of the modified image data and the second copy of themodified image data are simultaneously displayed within the output imageand separated by either an axis of symmetry or an axis of replication;and f) move the boundaries of the region of interest with respect to themedical image and repeat steps (c) through (e) so that as the outputimage changes with each movement of the boundaries of the region ofinterest, a position of a nodule appearing in the displayed first copyof the modified image data changes in a synchronized manner with respectto a position of the same nodule simultaneously appearing in thedisplayed second copy of the modified image data; and a display deviceconfigured to receive the output image and display the first copy of themodified image data simultaneously with the second copy of the modifiedimage data and separated by either the axis of symmetry or the axis ofreplication.
 7. The medical imaging system of claim 6, wherein thecomputer system is further configured to add padding pixels to aperimeter of the output image.
 8. The medical imaging system of claim 6,wherein the second copy of the modified image data includes a mirrorimage of the modified image data contained within the boundaries of theregion of interest.
 9. The medical imaging system of claim 6, whereinthe second copy of the modified image data includes a replicated imageof the modified image data contained within the boundaries of the regionof interest.
 10. The medical imaging system of claim 6, wherein thecomputer system is configured to move the region of interest a series oftimes; and wherein the display device is configured to display theoutput image as an animated display based on the series.
 11. The medicalimaging system of claim 6, wherein the computer system is configured tomodify the copy of the image data from within the boundaries of theregion of interest using the following equation:output=(input−A)*B+C where: output=value of a pixel in the modifiedimage data, input=value of a corresponding input pixel in the copy ofthe image data, A=a user defined constant, B=a user defined constant,C=a user defined constant.
 12. The medical imaging system of claim 11,wherein A equals a mean pixel value within a region around the inputpixel; wherein B equals 2.5; and wherein C equals a mean pixel value ofan entire input image times 0.9.
 13. The medical imaging system of claim12, wherein the region around the input pixel is centered over the inputpixel and has an area equal to a square of a width of the region ofinterest.
 14. The medical imaging system of claim 11, wherein wherein Aequals a minimum pixel value within the region around the input pixel;wherein B equals a maximum pixel value within a region around the inputpixel minus the minimum pixel value within the same region around theinput pixel, a sum of which is divided by a sum of the maximum pixelvalue of the input image minus the minimum pixel value of the inputimage; and wherein C equals the minimum pixel value of the input image.15. The medical imaging system of claim 11, wherein A is combined with avalue derived from pixel values in a first region around the inputpixel; wherein B is combined with a value derived from pixel values in asecond region around the input pixel; and wherein C is combined with avalue derived from pixel values in a third region around the inputpixel.
 16. A medical imaging system comprising: a computer system forreceiving input medical image data representing a medical image,metadata associated with the medical image data, and user commands, andfor processing the medical image data based upon the user commands toproduce a first copy of the medical image data and a second copy of themedical image data from within movable boundaries of a region ofinterest; and a display for receiving the first and second copies of themedical image data and displaying a representation of the movingboundaries of the region of interest containing the first copy of themedical image data simultaneously displayed with the second copy of themedical image data and separated by either an axis of symmetry or anaxis of replication so that a position of a nodule appearing in thedisplayed first copy of the medical image data changes in a synchronizedmanner with respect to a position of the same nodule simultaneouslyappearing in the displayed second copy of the medical image data as theboundaries of the region of interest move.
 17. The medical imagingsystem of claim 16, wherein the first copy of the medical image dataincludes original image data contained within the moving boundaries ofthe region of interest; and wherein the second copy of the medical imagedata includes a mirror image based on the medical image data within themoving boundaries of the region of interest.
 18. The medical imagingsystem of claim 16, wherein the first copy of the medical image dataincludes original image data contained within the moving boundaries ofthe region of interest; and wherein the second copy of the medical imagedata includes a replicated image based on the medical image data withinthe moving boundaries of the region of interest.
 19. The medical imagingsystem of claim 16, wherein the medical image data can be digitallyprocessed by the computer system with algorithms that enhance thedetectability of the nodules.